Existing Neural Architecture Search (NAS) methods either encode neural architectures using discrete encodings that do not scale well, or adopt supervised learning-based methods to jointly learn architecture representations and optimize architecture search on such representations which incurs search bias. Despite the widespread use, architecture representations learned in NAS are still poorly understood. We observe that the structural properties of neural architectures are hard to preserve in the latent space if architecture representation learning and search are coupled, resulting in less effective search performance. In this work, we find empirically that pre-training architecture representations using only neural architectures without their accuracies as labels improves the downstream architecture search efficiency. To explain this finding, we visualize how unsupervised architecture representation learning better encourages neural architectures with similar connections and operators to cluster together. This helps map neural architectures with similar performance to the same regions in the latent space and makes the transition of architectures in the latent space relatively smooth, which considerably benefits diverse downstream search strategies.

Weaknesses: Similar to this work, many recent NAS approaches perform architecture search in the continuous representation space, and gradient descent (GD) is one of the most commonly used approaches for architecture search in the continuous space. However, this work only considers RL and BO as the search algorithms. I am curious to know how competitive GD is compared to RL and BO. In Figure 4, I am not sure why the right plot has more blank space compared to the left plot. An explanation is needed to help the readers understand the insights behind those plots.

In the paper authors created an unsupervised learning method to embeds architectures in latent space and showed through experiments that the representations formed result in improved downstream performance compared to training with supervised objective jointly. The idea is important and the analysis is sound. The paper could be improved by analysing more diverse space of architectures than ResNet like blocks, as well as other suggestions given by the reviewers.

Existing Neural Architecture Search (NAS) methods either encode neural architectures using discrete encodings that do not scale well, or adopt supervised learning-based methods to jointly learn architecture representations and optimize architecture search on such representations which incurs search bias. Despite the widespread use, architecture representations learned in NAS are still poorly understood. We observe that the structural properties of neural architectures are hard to preserve in the latent space if architecture representation learning and search are coupled, resulting in less effective search performance. In this work, we find empirically that pre-training architecture representations using only neural architectures without their accuracies as labels improves the downstream architecture search efficiency. To explain this finding, we visualize how unsupervised architecture representation learning better encourages neural architectures with similar connections and operators to cluster together.